prime-runs - #2415
Draft
kcoopermiller wants to merge 23 commits into
Draft
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RpNehr64sniVfqKo62GKPw
`run_config()` now carries what the dashboard reads unconditionally — `model`, `num_examples` (the selected task count, passed in by the runner; -1 for "all" when unknown) and `rollouts_per_example` — alongside the `exclude_unset` dump, which drops defaults and so left the evals list showing "Unknown Model" and "0 examples". Credentials are stripped producer-side, so nothing sensitive reaches `metadata`: every value under a `headers` table and any value whose key names a credential (`api_key`, `Authorization`, `hf_token`; references such as `api_key_var` are kept) is masked in the structured dump, and the launched config file is uploaded verbatim only when parsing it (TOML/JSON) finds no credential-holding key — otherwise, or when it cannot be inspected, it is left out with a warning naming the offending paths. `open_run` honours `PRIME_RUNS_MODE=disabled`, which the explicit `mode="online"` used to override. A run that stays local keeps its own uuid instead of adopting the SDK's `disabled-…` placeholder. The E2E fixture sets `push=False`: `run_eval` opens a platform run before the first rollout when a key is present, and CI has one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyW4j57juAg5cheTX8PmUR
The only places an eval config can carry a credential are a client's `headers` table and a seat's `harness.env` table: the API key itself is `api_key_var` (an env-var name) and `forward_env` exists so harness secrets need not be written down. Mask everything under those two tables and skip the verbatim `config_source` when the file sets either, instead of guessing from key names. Drops the credential-key regex, its reference-key exemption list and the dotted-path scan. Also widen `_close(summary=)` to `Mapping` so `metrics.RunSummary` (a TypedDict) type-checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDrFiuDUjdJrv8Qu4oud8C
…on't flip import order Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HDEqDAXcNvbJ2GqsWttDhy
…ckouts don't flip import order" This reverts commit cca4860.
Upload only the three v0 metadata keys the dashboard reads (model, num_examples, rollouts_per_example), as the pre-SDK push did. The exclude_unset config dump and the launched file (config_source) are deferred to a follow-up: both can carry credentials (client.headers, harness.env), and the masking that made them safe was most of this module. This also keeps metadata.env a string, so the evals list no longer needs platform#4740 before this can ship. Removed: scrub_secrets, credential_tables and their helpers, RunConfig.source/record_source, and config_file_ref. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015bpEF1rKjCijybTYmfTGw2
Shorter docstrings and comments throughout; the PushState footer no longer reports per-sink losses (the SDK logs them); PushState is always constructed so the platform helpers take it unconditionally; run_config is inlined into open_run with num_examples required; build_samples is re-exported with a noqa instead of __all__. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015bpEF1rKjCijybTYmfTGw2
close() waited for in-flight cancel senders with `while self._cancel_tasks: await gather(*self._cancel_tasks)`. On 3.12+ a gather over tasks that are all already done completes eagerly without yielding to the loop, so the discard callbacks that empty the set never run and the coroutine spins at 100% CPU. It was latent on main, where close() is reached while the senders are still pending; the prime-runs integration waits for the rollouts to unwind first, so every Ctrl-C on a served eval hung. Gather only the pending tasks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015bpEF1rKjCijybTYmfTGw2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The eval CLI hand-rolled its own platform client:
verifiers/v1/utils/platform.pywas 317 lines of resolve env → create → post samples → finalize over raw httpx, run once after the last rollout. prime#856 moves that lifecycle intoprime-runs, a leaf SDK (httpx+prime-traces) where a run is an object held for the length of the eval rather than three stateless calls at the end of it.What
platform.pyis now ~120 lines of wiring; the SDK owns transport, retries, backpressure and the terminal status:Gone:
push_traces,credentials,json_bytes,run_metrics, the batching loop, the payload ceiling, the in-memory buffer.trace_to_sampleandbuild_samplesmoved toprime_runs.projection(knowledge about a platform wire format, not about an eval framework).build_samplesstays importable from here because prime-rl's released monitor imports it across the repo boundary; nothing importstrace_to_sample.utils/platform.pyopen_run/finish_run/abort_run+PushState, the dashboard's read-through view of the runcli/eval/runner.pygather_rolloutscancels sibling rollouts on the first failure so nothing keeps uploading into a run that is being closedcli/eval/main.pyconfigs/cli/eval.pyRunConfig.adopt_id()cli/dashboard/eval.pyPushing traces (<url>)while it streams, whiteTraces pushed (<url>), red on failuretests/v1/conftest.pypush=Falseserve/client.pyEnvClient.close()gathers only the still-pending cancel senders: on Python 3.12+ a gather over finished tasks completes without yielding, so the loop spun at 100% CPU on Ctrl-C once the runner started waiting for rollouts to unwind before teardownWhat the run carries
Only what the pre-SDK upload sent as the evaluation's
metadata:model,num_examples(the selected task count) androllouts_per_example, the v0 keys the dashboard's lists and reproduce command read.The config itself is deliberately not uploaded in this PR. The evaluations API is not beta-gated (only the Prime Traces sink is), and both a config dump and the launched
@ eval.tomlcan carry credentials (client.headers,harness.env), so uploading either needs producer-side masking. That and the platform's rendering of it (platform#4740, which rendersmetadata.config_sourceverbatim) land together in a follow-up once #4740 is deployed. Keepingmetadata.enva string here also means this PR does not depend on #4740.Modes
--no-pushkeeps the run local. So doesPRIME_RUNS_MODE=disabled, the SDK's own switch, honoured explicitly becauseopen_runpassesmode="online"and would otherwise override it. A local run keeps its own uuid rather than the SDK'sdisabled-…placeholder. A run that cannot be opened (no API key, hub resolution failure, a team outside an allowlist) falls back to local with the reason in the footer; the eval never fails because of the platform.Blocked on
prime-runson PyPI (prime#856). Until then the[tool.uv.sources]git pin (markedTEMPORARY) stays, published verifiers wheels are uninstallable withpip, and the sandbox-side installs in the live E2E cannot resolveprime-runs>=0.1.0, which is the current red E2E job. Once it is live: drop the pin,uv lock, and the E2E should go green.